Towards Increasing Reliability of Heuristic Evaluation

To date comparative studies have not convinced the HCI community about the reliability of results from usability evaluations. User testing of the same products by different usability teams found different problems (Molich et al., 1999). Discount methods also have the same problem. With heuristic evaluation method, for example, overlaps between evaluators’ results are often as low as 10% (Molich & Robin, 2003). However, the strength of the heuristic evaluation method is in its cost-effectiveness. In a comparative study (Cuomo & Bowen, 1994) of heuristic evaluation, cognitive walkthrough, and interface guidelines, heuristic evaluation found the highest percentage of true problems per hour of inspection (9.3% as compared to 6.8% by cognitive walkthrough and 2.3% by guidelines). Furthermore, there is a pressing need for a discount evaluation method like heuristic evaluation. As a ‘quick and easy to perform’ method, heuristic evaluation was rated as the most commonly used evaluation method by 134 HCI practitioners at CHI 98, CHI 99, and UPA 99 Conferences (Rosenbaum et al., 2000). However, to best benefit from the method, we need to find ways to improve the reliability of its results.